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Abstract. We extend the statistical neurodynamics to study transient 
dynamics of sequence processing neural networks with finite dilution, 
and the theoretical results are supported by extensive numerical simu- 
lations. It is found that the order parameter equations are completely 
equivalent to those of the Generating Functional Method, which means 
that crosstalk noise follows normal distribution even in the case of fail- 
ure in retrieval process. In order to verify the gaussian assumption of 
crosstalk noise, we numerically obtain the cumulants of crosstalk noise, 
and third- and fourth-order cumulants are found to be indeed zero even 
in non-retrieval case. 



1 Introduction 

Models of attractor neural networks for processing sequences of patterns, as a 
realization of a temporal association, have been of great interest over some time 
[1-5]. On contrary to Hopfield model [6,7], the asymmetry of the interaction 
matrix in this model leads to violation of detailed balance, ruling out an equi- 
librium statistical mechanics analysis. Usually, Generating Functional Method 
and Statistical Neurodynamics are two ways to study this model. Generating 
Functional Method [2, 13-15] is also known as e.g. path integral method, and 
it allows the exact solution of the dynamics and to search all relevant physical 
order parameters at any time step via the derivatives of a generating functional. 
Statistical Neurodynamics (so called Signal- To-Noise Analysis) [3, 8, 9, 12], starts 
from splitting of the local field into a signal part originating from the pattern 
to be retrieved and a noise part arising from the other patterns, and uses sim- 
ple assumption that crosstalk noise is normally distributed with zero mean and 
evaluated variance. In Hopfield model, this treatment is proved to be just an ap- 
proximation and the Gaussian form of crosstalk noise only holds when pattern 
recall occurs [10,11,17]. But Kawamuran et al. suggest that Statistical Neuro- 
dynamics is exact in fully connected sequence processing model [3] . In this paper 
we show that treatment of Statistical Neurodynamics is also exact correct in the 
case of diluted connection. 
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In the nature of brain, fully-connected structure of network is biological un- 
realistic. Therefore, as a relaxation of conventional, unrealistic condition that 
every neuron is connected to every other, diluted networks have received lots 
of attention [18-22]. There are several kinds of dilution, with different aver- 
age connections cN, where c is connection probability. If cN = O(lnn), it is 
called extreme dilution, and in this case, network has a local Caley-Tree struc- 
ture and almost all pairs of neurons have entirely different sets of ancestors. 
Thus, the correlations in noise may be neglected. The reported first that the 
extremely diluted asymmetric network can be calculated exactly is Derrida et 
al [19]. Extremely diluted symmetric Hopfield network was also studied using 
replica- symmetric calculation. When cN = 0(1), network only have finite con- 
nection regardless of its scale, and the equilibrium properties has been calculated 
using replica-symmetric approximation [23]. In this work, we focus on the case 
that cN = 0(N), and we will show that our theoretical result also suitable 
for extremely diluted network cN <C N. Recently, Theumann [4] studied this 
type of dilution using Generating Functional Method, and we will discuss the 
relationship between our result and those obtained by Theumann. 

This paper was organized as follows. In Section 2 we recall the sequence 
processing neural networks with finite dilution. In Section 3 we use statistical 
neurodynamics to obtain the order parameter equations which describe dynam- 
ics of network, and compare our theoretical result with numerical simulations. 
Section 4 discuss the relationship between our results and those obtained by 
Generating Functional Method. Section 5 contains conclusion and discussion. 

2 Model definition 

Let us consider a sequence processing model that consists of N spins of neurons, 
when TV — > oo. The state of spins takes Si (t) = ±1 and updates the state 
synchronously with following probability [24] : 

Prob[Si (t + 1) \hi (t)] = ^[l + Si(t + l)tanhphi(t)}, (1) 

where /3 = 1/T is inverse temperature and local field is given by 

N 

M*) = E J ^W- ( 2 ) 

When we use F (•) to express transfer function, the parallel dynamics is expressed 

by 

Si(t + l) = F(hi(t)). (3) 

We store P = aN random patterns £ M = in network, where a 

is the loading ratio. Interaction matrix is chosen to retrieve the patterns as 
— > £ 2 — > ...£ p — > I; 1 sequentially. For instance, it is given by [5] 
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with £( p+1 ) = dj takes its value as or 1 probabilities 

Prob (dj = 1) = 1 - Prob (c y = 0) = c (5) 

with (0 < c < 1) and aj = Cji for symmetric dilution. 

Let us consider the case to retrieve £ 9 . We define m q (t) as the overlap pa- 
rameter between the network state S (t) and the condensed pattern £ q 

1 N 

m'(*) = Tf (6) 



3 Statistical Neurodynamic for diluted networks 

We start from Somplinsky's idea that in the limit N — > oo the synaptic matrix 
in (4) can be written as a fully connected model with synaptic noise [22] 

^r = ^E^+^ (7) 

where rjij is a complex random variable following Gaussian distribution with 
mean and variance a(l — c)/c. Theumann also proved this relationship by 
taking expansion of averaging e - 4h (*) JS (*) 0V er disorder, where h is auxiliary 
local field (see section 4). 

Then local field is decribed by 

hi (t) = tf +1 m q (t) + Z t (t) . (8) 

Here Zi (t) denotes the crosstalk noise term from uncondensed patterns, 

1 aN N N 

2.W^EE Z^tjSi (*) + E s > (*) m • ( 9 ) 

We assume that the noise term Zi (t) is normally distributed with mean 
and variance of (Zi (t)). For Hopfield model, the assumption is shown to be 
valid within statistical errors by Monte Carlo simulations as long as the memory 
retrieval is successful [10]. And for sequence processing neural networks, the 
assumptions is shown to be hold even in the non-retrieval case [3] . We also keep 
this assumption in our model with presence of Cij , which is verified by numerical 
simulations of cumulants of Zi (t) . 

To obtain the variance a 2 (Zi (t + 1)) in a recursive form, we express the 
noise term of the transfer function F (•), 

-. aN N N 

Z =( t+1 ) = ^EE tf +1 $ F fa (*)) + E F fa (*)) % 

N 

= Y l (t + l)+Y J F(h (t)) mo , (10) 
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where mean of Y{ (t + 1) is zero, and 

N aN 

hi (t) = ^m*- 1 (t)+jv£ E (*-l) + E 5 *(*- !) ^ (") 

fc#j "#9-1 J 

Since /ij (i) depends strongly on and the term X^fe^j ^j^k~ 1 ^k (t) can be 
rationally considered as stochastic order 0(1/ \/N), one can expand the transfer 
function F (•) up to the first order, then obtain 

Y * (* + x ) = ^ E E w) + (*) . (12) 



where 



AT aiV 



N 



h j (t) = e j m"- i (t) + ^Y j e er^^-^+E^c*- 1 )^ ( 13 ) 

and 



1 W 

^ <+1 ) = ^E F ' (M*))- 



(14) 



Since crosstalk noise is assumed to be zero mean, one can calculate variance 
of crosstalk noise directly by 



a 2 (Zi (t + 1)) = a 2 (Yi (t + 1)) + a(l - c)/c. 
Firstly, we have to determine a 2 (Yi (t + 1)), 



(15) 



1 



a 2 (Y t (t+l)) = —E 



N 2 
+ E 



EE(r)(^) 2 WMt))) : 



3 M 



+ [/ 2 (t + l)a 2 (Y t (t)) 



E E tf+^r 1 ^ (t + 1) sr ^ w 



j,k n,v 



(16) 



where we use Sj (t+1) = F (hj (t)^. According to the literature [3, 12], when 
expand Sk (t) up to t = 0, the last term in Eq. (16) becomes 



N 



-E 



t+l / n 



e n u (* + 1 - t ) e s i (* + !) ^ (* + 1 - ») e w 



n— 1 \r— 1 



(17) 



Here all terms in Eq. (17) are independent of each other except for n = aN, 2aN, 3aN.... 
So when N — > oo, the dependence vanishes, and the last term in Eq. (16) is ig- 
nored. The variance of Yi (t + 1) term becomes 



a 2 (Y { ( t +l)) = a+ U 2 (t + 1) af (Y t (t)) . 



(18) 
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When variance of crosstalk noise is determined using Eqs. (15, 18), all order 
parameters can be expressed by the following closed equations 

m (t + 1) = J Dz (£F (£m (t) + a (Z t (t)) z))^ , (19) 

u {t + 1} = <r(z!(t)) / Dzz {F (em {t) + ° {Zi {t)) z)> « ' (2o) 

a 2 {Zi (t)) = ^ + t/ 2 (t) (a 2 [Zi (t - 1)) - a (1 - c) /c) , (21) 

where (-)j stands for the average over the retrieval pattern £, and = -^=ea;p (— z 2 /2) dz. 

In the case that the temperature is absolute zero, F (■) = sgn(-), we get 
following expressions of order parameters 

m(t + l) = erff ?® ), (22) 



where 



a(Z i (t)) / 

u(t+i)= _,: u „ \i^(- "~ jTT, ) , m 




erf (u) = v 2/-7T / exp (-x 2 /2) dx. (24) 



'0 



This finishes statistical neurodynamics treatment of Sequence Processing 
Neural Networks with finite synaptic dilution. The above equations form a re- 
cursive scheme to calculate the dynamical properties of the systems with an 
arbitrary time step. The time evolution of overlaps obtained both in theory and 
numerical simulation are plotted in Fig. 1. Initial overlaps range from 0.1 to 1.0. 
when initial overlap is smaller than 0.5, network will fail to retrieval, and overlap 
will finally vanishes. Fig. 2 shows the basin of attraction both in theory and in 
numerical simulations. 



4 Generating Functional Method and Statistical 
Neurodynamics 

The idea of generating functional method is to concentrate on the moment gen- 
erating function Z [ip], which fully captures the statistics of paths, 

ZM= E ^^(0),...^(i)]e- l ^< t S(s) ^ (s) . (25) 

(7(0),..., CT (t) 

The generating function Z [tp] involves the overlap parameter m (t) . The response 
functions G (t, t') and the correlation functions C (t, t') are 
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Fig. 1. Temporal evolution of overlap m(t), initial overlap ranging from 1.0 to 0.1 (up 
to down). The parameters of networks are N = 5000, c = 0.2 and a/c = 0.38. 
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Fig. 2. Basin of attraction obtained by theory and numerical simulations. The param- 
eters of networks are N = 5000 and c = 0.1. 
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Diiring et al. first discussed the sequence processing model using Generating 
Functional Method and obtained dynamical equations in the form of a multiple 
Gaussian integral [2], which is too complex to calculate. Then Kawamura et 
al. simplified those equations and obtained a tractable description of dynamical 
equations with single Gaussian integral [3] . Theumann discussed the case of finite 
dilution using Generating Functional Method [4] , with idea that 

( e -ih( s )-J eff S(t)^ = e -ih(t)-JS(t) e -zA 2 h 2 (t)/2 ; ( 29 ) 

where A 2 = a (1 — c) /c presents variance of independent Gaussian random vari- 
ables, which is exactly the idea of Sompolinsky that Jff* = + [22]. With 
this formula, Theumann obtained the temporal evolution of order parameters 
using the same scheme to that in [2,3], 

m(t) = J Dz (tanh/3 m (t - 1) + 6 (t - 1) + zy/aD (t - l,t- 1) ^ , (30) 



G(t,t- 1) =/3 jl -J Dz(tanh 2 /3 m (t - 1) + 9 (t - 1) + z^/aD (t - 1, t - 1) ^ j , 

(31) 

and C (t,t) = 1, where Dz = 1 /V2tt cxp (— z 2 /2) . Covariance matrix of crosstalk 
noise is given by 

D(t,t) = R(t,t) + (l-c)/c, (32) 

R (t, t) = 1 + G 2 (t, t - 1) R (t - 1, t - 1) . (33) 

From Eq. (19) to Eq. (30), Note that a 2 (Z, L (t)) corresponds to aD(t,t'), 
and U (t) corresponds to G(t,t — 1). It is easy to find that Statistical Neuro- 
dynamics and Generating Functional Method present the same order parameter 
equations of temporal evolution. It means that the Gaussian form of crosstalk 
noise holds and Statistical Neurodynamics can give the exact solution, compar- 
ing with Hopfield model that the crosstalk noise is normally distributed only in 
retrieval case [10]. To verify the distribution of crosstalk noise, the first, second, 
third, and fourth cumulants c\ (t) , C2 (t) , c 3 (t) , c 4 (i) are evaluated numerically, 
and the third and fourth cumulants are found to be zero even when network fails 
in retrieval (see Fig. 3). 



5 Conclusion and discussion 



In this paper, Statistical Neurodynamics is extended to study the retrieval dy- 
namics of sequence processing neural networks with random synaptic dilution. 
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Fig. 3. Temporal evolution of cumulants C\ (t) , C2 (t) , Cz (t) , C4 (t), and overlap m (i). 
The initial overlap is m (0) = 0.2. The parameters of networks are N = 5000, c = 0.06, 
and a/c — 0.5. 



Our theoretical results are exactly consistent with numerical simulations. The 
order parameter equations are obtained which is complete equivalent to those 
obtained by Generating Functional Method. We also present the first, second, 
third and forth-order cumulants of crosstalk noise to verify the Gaussian distri- 
bution of noise. 

Finally, note that, in fully connected network or extremely diluted network, 
one can also obtain the order parameter equations from our Eqs. (19-21). For c = 
1, a 2 (Zi (t)) = ex 2 (Yi (t)) and equation (21) corresponds to the order parameter 
equations in fully connected networks [3]. For the limit cN <C N, a <C a/c, then 
the variance of crosstalk noise is always a/c, that is exactly the result obtained 
in extremely diluted network where the local Cayley-Tree structure is held and 
all correlations in noise are neglected [19]. 
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